Simultaneous Localization And Mapping Research Articles

Tracking and co-location of global point clouds for large-area indoor environments

Extended reality (XR) experiences are on the verge of becoming widely adopted in diverse application domains. An essential part of the technology is accurate tracking and localization of the headset to create an immersive experience. A subset of the applications require perfect co-location between the real and the virtual world, where virtual objects are aligned with real-world counterparts. Current headsets support co-location for small areas, but suffer from drift when scaling up to larger ones such as buildings or factories. This paper proposes tools and solutions for this challenge by splitting up the simultaneous localization and mapping (SLAM) into separate mapping and localization stages. In the pre-processing stage, a feature map is built for the entire tracking area. A global optimizer is applied to correct the deformations caused by drift, guided by a sparse set of ground truth markers in the point cloud of a laser scan. Optionally, further refinement is applied by matching features between the ground truth keyframe images and their rendered-out SLAM estimates of the point cloud. In the second, real-time stage, the rectified feature map is used to perform localization and sensor fusion between the global tracking and the headset. The results show that the approach achieves robust co-location between the virtual and the real 3D environment for large and complex tracking environments.

Viewpoint-Aware Visibility Scoring for Point Cloud Registration in Loop Closure

Viewpoint-Aware Visibility Scoring for Point Cloud Registration in Loop Closure

PSMD-SLAM: Panoptic Segmentation-Aided Multi-Sensor Fusion Simultaneous Localization and Mapping in Dynamic Scenes

Multi-sensor fusion is pivotal in augmenting the robustness and precision of simultaneous localization and mapping (SLAM) systems. The LiDAR–visual–inertial approach has been empirically shown to adeptly amalgamate the benefits of these sensors for SLAM across various scenarios. Furthermore, methods of panoptic segmentation have been introduced to deliver pixel-level semantic and instance segmentation data in a single instance. This paper delves deeper into these methodologies, introducing PSMD-SLAM, a novel panoptic segmentation assisted multi-sensor fusion SLAM approach tailored for dynamic environments. Our approach employs both probability propagation-based and PCA-based clustering techniques, supplemented by panoptic segmentation. This is utilized for dynamic object detection and the removal of visual and LiDAR data, respectively. Furthermore, we introduce a module designed for the robust real-time estimation of the 6D pose of dynamic objects. We test our approach on a publicly available dataset and show that PSMD-SLAM outperforms other SLAM algorithms in terms of accuracy and robustness, especially in dynamic environments.

Lp-slam: language-perceptive RGB-D SLAM framework exploiting large language model

Lp-slam: language-perceptive RGB-D SLAM framework exploiting large language model

Action-Aware Vision Language Navigation (AAVLN): AI vision system based on cross-modal transformer for understanding and navigating dynamic environments

Visually impaired individuals face great challenges with independently navigating dynamic environments because of their inability to fully comprehend the environment and actions of surrounding people. Conventional navigation approaches like Simultaneous Localization And Mapping (SLAM) rely on complete scanned maps to navigate static, fixed environments. With Vision Language Navigation (VLN), agents can understand semantic information to expand navigation to similar environments. However, both cannot accurately navigate dynamic environments containing human actions. To address this challenge, we propose a novel cross-modal transformer-based Action-Aware VLN system (AAVLN). Our AAVLN Agent Algorithm is trained using Reinforcement Learning in our Environment Simulator. AAVLNs novel cross-modal transformer structure allows the Agent Algorithm to understand natural language instructions and semantic information for navigating dynamic environments and recognizing human actions. For training, we use Reinforcement Learning in our action-based environment simulator. We created it by combining an existing simulator with our novel 3D human action generator. Our experimental results demonstrate the effectiveness of our approach, outperforming current methods on various metrics across challenging benchmarks. Our ablation studies also highlight that we increase dynamic navigation accuracy with our Vision Transformer based human action recognition module and cross-modal encoding. We are currently constructing 3D models of real-world environments, including hospitals and schools, for further training AAVLN. Our project will be combined with Chat-GPT to improve natural language interactions. AAVLN will have numerous applications in robotics, AR, and other computer vision fields.

Bundle adjustment with motion constraints for uncalibrated multi-camera systems at the ground level

Multi-camera systems for structure from motion (SfM) are widely deployed in many mapping applications. Existing solutions assume known rig calibration, synchronized frames among cameras, as well as overlapping field of views (FoVs). In this paper, we derive novel geometric constraints assuming minimal knowns about the multi-camera systems, to benefit low-cost and non-expert use cases where uncalibrated multi-camera systems with non-typical geometry setups present, i.e., no rig calibration, no overlapping FoVs. Assuming that these cameras are co-located and share the same motion of the platform, the proposed constraints utilize the parallelism and length proportionality of motion vectors of these co-located cameras and formulate them as translation constraints into the bundle adjustment (BA). The proposed constraints (called motion constraints) impose a first-order penalty to co-located cameras whose motion speeds and directions between frames do not match. With soft constraints, it can handle loosely synchronized frames (with an error within one second). The proposed constraints are integrated into the BA framework and experimented with different camera setups, i.e., on a group of casually co-located GoPro cameras with no rig calibration, and some with no overlapping views. Our results show that the constraints are extremely effective in improving the reconstruction and pose accuracy for ground motion images: in our self-collected open trajectories without loop closure, the proposed constraints are effective in correcting topographical errors (i.e., trajectory drifts) of the resulting models, and the dense point clouds achieve up to 11.34 m (86.12 %) of mean absolute error (MAE) improvement as compared to reference LiDAR point clouds; our results on KITTI-odometry and KITTI-360 datasets also show an improvement of up to 28.82 m (81.05 %) in terms of the root mean square error (RMSE) of absolute pose error (APE). We expect that the proposed constraints are significant not only as additional geometric constraints for image-based mobile mapping, but also will benefit the broader use of photogrammetry, since it empowers the possibility to harness the traditionally so-called low-quality stereo/multi-camera data (e.g., by non-photogrammetry citizen scientists) into improved 3D products.

Monomeric Esterase: Insights into Cooperative Behavior, Hysteresis/Allokairy.

Herein, we present a novel esterase enzyme, Ade1, isolated from a metagenomic library of Amazonian dark earths soils, demonstrating its broad substrate promiscuity by hydrolyzing ester bonds linked to aliphatic groups. The three-dimensional structure of the enzyme was solved in the presence and absence of substrate (tributyrin), revealing its classification within the α/β-hydrolase superfamily. Despite being a monomeric enzyme, enzymatic assays reveal a cooperative behavior with a sigmoidal profile (initial velocities vs substrate concentrations). Our investigation brings to light the allokairy/hysteresis behavior of Ade1, as evidenced by a transient burst profile during the hydrolysis of substrates such as p-nitrophenyl butyrate and p-nitrophenyl octanoate. Crystal structures of Ade1, coupled with molecular dynamics simulations, unveil the existence of multiple conformational structures within a single molecular state (E̅1). Notably, substrate binding induces a loop closure that traps the substrate in the catalytic site. Upon product release, the cap domain opens simultaneously with structural changes, transitioning the enzyme to a new molecular state (E̅2). This study advances our understanding of hysteresis/allokairy mechanisms, a temporal regulation that appears more pervasive than previously acknowledged and extends its presence to metabolic enzymes. These findings also hold potential implications for addressing human diseases associated with metabolic dysregulation.

ACCURACY ASSESSMENT OF A SLAM-ACQUIRED POINT CLOUD DATA USING A VARIETY OF CLASSIFICATION APPLICATIONS

Abstract. Laser scanning techniques, such as Simultaneous Localization and Mapping (SLAM), produce three-dimensional data representing the real world, which may provide significant information for Building Information Models (BIM). These processes produce 3D point clouds, which require classification before being used in various applications such as structural assessments. However, most widely available software applications for classifying 3D point clouds are proprietary, giving an incomplete depiction of how the data is manipulated and processed. Thus, this research aims to assess the accuracy of the different classification applications in classifying the 3D point cloud data and perform a comparative analysis of the results. Precision, Recall, F1-score, and Accuracy are the evaluation metrics used to assess the classified 3D point cloud data. Results for Precision and Recall show that some of the applications can classify a particular class, the Ground and Building classes. However, the overall performance of the classification method, which is evaluated through the F1-score, produced low values. Results for the F1-score demonstrate that these low values indicate low overall reliability of the classification results despite high values for Accuracy. Based on the conducted experiments, further research is suggested to investigate the effect of increasing dataset size and equalizing class sizes used in classification.

MF-LIO: Integrating Multi-Feature LiDAR Inertial Odometry with FPFH Loop Closure in SLAM

Abstract Simultaneous Localization and Mapping (SLAM) is a pivotal technology in autonomous vehicle navigation and a significant research area in robotics. Addressing the inaccuracies in point cloud registration of traditional LiDAR SLAM, which lead to localization and mapping errors, we propose a novel LiDAR Inertial Odometry approach integrating IMU and a multi-feature joint registration strategy. Initially, we introduce an innovative ground segmentation method and feature categorization strategy, enhancing ground detection mechanisms and optimizing the feature extraction process. Subsequently, our multi-feature joint registration method computes the pose transformations between current frames and the local map. Finally, we employ a global registration method based on Fast Point Feature Histograms (FPFH) feature descriptors for coarse alignment, providing initial estimates for the Generalized Iterative Closest Point (GICP) algorithm, thus efficiently and accurately mitigating cumulative errors. Extensive evaluations on the KITTI dataset and real-world campus environment demonstrate that our approach significantly surpasses existing advanced LiDAR SLAM solutions, achieving over a 24% improvement in pose estimation accuracy.

2DLIW-SLAM:2D LiDAR-inertial-wheel odometry with real-time loop closure

Due to budgetary constraints, indoor navigation typically employs two-dimensional (2D) LiDAR rather than 3D LiDAR. However, the utilization of 2D LiDAR in simultaneous localization and mapping (SLAM) frequently encounters challenges related to motion degeneracy, particularly in geometrically similar environments. To address this problem, this paper proposes a robust, accurate, and multi-sensor-fused 2D LiDAR SLAM system specifically designed for indoor mobile robots. To commence, the original LiDAR data undergoes meticulous processing through point and line extraction. Leveraging the distinctive characteristics of indoor environments, line–line constraints are established to complement other sensor data effectively, thereby augmenting the overall robustness and precision of the system. Concurrently, a tightly-coupled front-end is created, integrating data from the 2D LiDAR, inertial measurement unit, and wheel odometry, thus enabling real-time state estimation. Building upon this solid foundation, a novel global feature point matching-based loop closure detection algorithm is proposed. This algorithm proves highly effective in mitigating front-end accumulated errors and ultimately constructs a globally consistent map. The experimental results indicate that our system fully meets real-time requirements. When compared to cartographer, our system not only exhibits lower trajectory errors but also demonstrates stronger robustness, particularly in degeneracy problem. We open source our methods here: https://github.com/LittleDang/2DLIW-SLAM.

3D RECONSTRUCTION OF THE FORT SANTIAGO DUNGEONS USING HANDHELD LASER SCANNING METHOD

Abstract. The preservation of Cultural Heritage sites is an inherent responsibility of the citizens of a country. To this end, developments in digitization serve as a permanent way of preserving a site at its optimum state, and the creation of accurate 3D models becomes a repository of measurements for future reconstruction. Thus, in this study, the Fort Santiago Dungeons, a popular Cultural Heritage site in the Philippines, was 3D-reconstructed using the GeoSLAM ZEB Revo RT, a handheld Light Detection and Ranging (LiDAR) scanner that uses a Simultaneous Localization and Mapping (SLAM) Algorithm. With it, dense point clouds were produced with acceptable results in terms of colorization using the ZEB-CAM accessory, completeness, and measurement noise. This trend extended to quantitative evaluations of the dense point clouds of different formats (LAS and PLY) and colorization settings, as the Root Mean Square Errors (RMSE) against corresponding ground measurements were 0.734cm and 0.739cm for the Non-colorized, and 1.308cm and 1.312cm for Colorized, respectively. Likewise, the derived mesh files gave similar RMSEs with the Non-colorized at 1.044cm (LAS) and 0.999cm (PLY); and with the Colorized at 1.339cm (LAS) and 1.326cm (PLY). Overall, results showed that the mesh file derived from PLY dense point cloud exceeded the other format - as the dungeon's interior was better represented. Nonetheless, even though there were differences in the RMSEs of the measurements, due to factors such as the number of points and the processing itself, all are within the 5cm threshold to be considered useful in the event of reconstruction.

LGD: A fast place recognition method based on the fusion of local and global descriptors

Place recognition, which is also known as loop detection, has been one of the main problems in the field of robot navigation, and it is crucial in realizing high-performance simultaneous localization and mapping (SLAM). Most of the existing methods directly construct local or global descriptors with point clouds, ignoring the offset of different visits in the same place. To overcome the current limitations, this study emphasizes the importance of point cloud features in improving the environment recognition precision and proposes an innovative framework that combines local features that are spatially invariant and prominent descriptive global features as a representation of a laser scanner frame. The feature center of the laser point cloud is obtained by computing local feature points with spatial invariance, which improves the offset between the original point cloud and the feature center. Furthermore, a coarse-to-fine hierarchical recognition strategy is developed to improve the efficiency of place recognition. An evaluation with different offsets and synthesis is conducted on the publicly available KITTI and self-developed CHDloop datasets. The experimental results show that the proposed method can improve time efficiency by more than 18% compared to the SC and ISC, and there are also significant improvements in the recall rate and accuracy in slight lane-change scenarios. Moreover, the proposed method can obtain rich lane-change loops in moderate and large lane-change scenarios where the SC and ISC fail to obtain effective loops. The proposed method can be used as a lightweight and interpretable approach for place recognition, which not only can be quickly deployed in any SLAM framework but can also efficiently handle more complex place recognition issues in urban environments.

Robust loop closure detection and relocalization with semantic-line graph matching constraints in indoor environments

Loop closure detection (LCD) plays an essential role in the Simultaneous Localization and Mapping (SLAM) process, effectively reducing cumulative trajectory errors. However, conventional LCD methods often encounter challenges when dealing with variations in illumination, changes in viewpoint, and environments with weak textures. This is due to their reliance on low-level geometric or image features. To address these issues, we propose a robust LCD method named SL-LCD, which integrates semantic information and line features to fully leverage the semantic content and line attributes within indoor scenes, thereby establishing a reliable feature correspondence between query images and loop closure images. For the retrieval of candidate closed-loop images, we construct a semantic-line-segment topological graph and introduce a graph matching algorithm to perform the LCD task. This approach fully exploits image features and spatial information to achieve closed-loop detection in complex indoor scenes. Furthermore, we present a semantic voxel-based generalized ICP (SVGICP) closed-loop relocalization algorithm tailored for challenging and complex indoor scenes, enhancing the accuracy of closed-loop relocalization in such scenarios. Experimental results demonstrate that the SL-LCD algorithm proposed in this paper surpasses state-of-the-art methods, accurately detecting closed loops, and effectively eliminating trajectory drift.

Slam loop closure detection algorithm based on MSA-SG

Slam loop closure detection algorithm based on MSA-SG

MDP-SLAM: A Visual SLAM towards a Dynamic Indoor Scene Based on Adaptive Mask Dilation and Dynamic Probability

Visual simultaneous localization and mapping (SLAM) algorithms in dynamic scenes will apply the moving feature points to the camera pose’s calculation, which will cause the continuous accumulation of errors. As a target-detection tool, mask R-CNN, which is often used in combination with the former, due to the limited training datasets, easily results in the semantic mask being incomplete and deformed, which will increase the error. In order to solve the above problems, we propose in this paper a visual SLAM algorithm based on an adaptive mask dilation strategy and the dynamic probability of the feature points, named MDP-SLAM. Firstly, we use the mask R-CNN target-detection algorithm to obtain the initial mask of the dynamic target. On this basis, an adaptive mask-dilation algorithm is used to obtain a mask that can completely cover the dynamic target and part of the surrounding scene. Then, we use the K-means clustering algorithm to segment the depth image information in the mask coverage area into absolute dynamic regions and relative dynamic regions. Combined with the epipolar constraint and the semantic constraint, the dynamic probability of the feature points is calculated, and then, the highly dynamic possible feature points are removed to solve an accurate final pose of the camera. Finally, the method is tested on the TUM RGB-D dataset. The results show that the MDP-SLAM algorithm proposed in this paper can effectively improve the accuracy of attitude estimation and has high accuracy and robustness in dynamic indoor scenes.

LTA‐OM: Long‐term association LiDAR–IMU odometry and mapping

AbstractThis paper focuses on the Light Detection and Ranging (LiDAR)–Inertial Measurement Unit (IMU) simultaneous localization and mapping (SLAM) problem: How to fuse the sensor measurement from the LiDAR and IMU to online estimate robot's poses and build a consistent map of the environment. This paper presents LTA‐OM: an efficient, robust, and accurate LiDAR SLAM system. Employing fast direct LiDAR‐inertial odometry (FAST‐LIO2) and Stable Triangle Descriptor as LiDAR–IMU odometry and the loop detection method, respectively, LTA‐OM is implemented to be functionally complete, including loop detection and correction, false‐positive loop closure rejection, long‐term association (LTA) mapping, and multisession localization and mapping. One novelty of this paper is the real‐time LTA mapping, which exploits the direct scan‐to‐map registration of FAST‐LIO2 and employs the corrected history map to provide direct global constraints to the LIO mapping process. LTA mapping also has the notable advantage of achieving drift‐free odometry at revisit places. Besides, a multisession mode is designed to allow the user to store the current session's results, including the corrected map points, optimized odometry, and descriptor database for future sessions. The benefits of this mode are additional accuracy improvement and consistent map stitching, which is helpful for life‐long mapping. Furthermore, LTA‐OM has valuable features for robot control and path planning, including high‐frequency and real‐time odometry, driftless odometry at revisit places, and fast loop closing convergence. LTA‐OM is versatile as it is applicable to both multiline spinning and solid‐state LiDARs, mobile robots and handheld platforms. In experiments, we exhaustively benchmark LTA‐OM and other state‐of‐the‐art LiDAR systems with 18 data sequences. The results show that LTA‐OM steadily outperforms other systems regarding trajectory accuracy, map consistency, and time consumption. The robustness of LTA‐OM is validated in a challenging scene—a multilevel building having similar structures at different levels. To demonstrate our system, we created a video which can be found on https://youtu.be/DVwppEKlKps.

A framework for collaborative multi-robot mapping using spectral graph wavelets

The exploration of large-scale unknown environments can benefit from the deployment of multiple robots for collaborative mapping. Each robot explores a section of the environment and communicates onboard pose estimates and maps to a central server to build an optimized global multi-robot map. Naturally, inconsistencies can arise between onboard and server estimates due to onboard odometry drift, failures, or degeneracies. The mapping server can correct and overcome such failure cases using computationally expensive operations such as inter-robot loop closure detection and multi-modal mapping. However, the individual robots do not benefit from the collaborative map if the mapping server provides no feedback. Although server updates from the multi-robot map can greatly alleviate the robotic mission strategically, most existing work lacks them, due to their associated computational and bandwidth-related costs. Motivated by this challenge, this paper proposes a novel collaborative mapping framework that enables global mapping consistency among robots and the mapping server. In particular, we propose graph spectral analysis, at different spatial scales, to detect structural differences between robot and server graphs, and to generate necessary constraints for the individual robot pose graphs. Our approach specifically finds the nodes that correspond to the drift’s origin rather than the nodes where the error becomes too large. We thoroughly analyze and validate our proposed framework using several real-world multi-robot field deployments where we show improvements of the onboard system up to 90% and can recover the onboard estimation from localization failures and even from the degeneracies within its estimation.

SG-ISBP: Orchard Robots Localization and Mapping With Ground Optimization and Loop Closure Detection Integration

SG-ISBP: Orchard Robots Localization and Mapping With Ground Optimization and Loop Closure Detection Integration

High-recall calibration monitoring for stereo cameras

Cameras are the prevalent sensors used for perception in autonomous robotic systems, but their initial calibration may degrade over time due to dynamic factors. This may lead to a failure of downstream tasks, such as simultaneous localization and mapping (SLAM) or object recognition. Hence, a computationally lightweight process that detects the decalibration is of interest. We describe a modification of StOCaMo, an online calibration monitoring procedure for a stereoscopic system. The method uses robust kernel correlation based on epipolar constraints; it validates extrinsic calibration parameters on a single frame with no temporal tracking. In this paper, we present a modified StOCaMo with an improved recall rate on small decalibrations through a confirmation technique based on resampled variance. With fixed parameters learned on a realistic synthetic dataset from CARLA, StOCaMo and its proposed modification were tested on multiple sequences from two real-world datasets: KITTI and EuRoC MAV. The modification improved the recall of StOCaMo by 25 % (to 91 % and 82 %, respectively), and the accuracy by 12 % (to 94.7 % and 87.5 %, respectively), while labeling at most one-third of the input data as uninformative. The upgraded method achieved the rank correlation between StOCaMo V-index and downstream SLAM error of 0.78 (Spearman).

A Compact Handheld Sensor Package with Sensor Fusion for Comprehensive and Robust 3D Mapping.

This paper introduces an innovative approach to 3D environmental mapping through the integration of a compact, handheld sensor package with a two-stage sensor fusion pipeline. The sensor package, incorporating LiDAR, IMU, RGB, and thermal cameras, enables comprehensive and robust 3D mapping of various environments. By leveraging Simultaneous Localization and Mapping (SLAM) and thermal imaging, our solution offers good performance in conditions where global positioning is unavailable and in visually degraded environments. The sensor package runs a real-time LiDAR-Inertial SLAM algorithm, generating a dense point cloud map that accurately reconstructs the geometric features of the environment. Following the acquisition of that point cloud, we post-process these data by fusing them with images from the RGB and thermal cameras and produce a detailed, color-enriched 3D map that is useful and adaptable to different mission requirements. We demonstrated our system in a variety of scenarios, from indoor to outdoor conditions, and the results showcased the effectiveness and applicability of our sensor package and fusion pipeline. This system can be applied in a wide range of applications, ranging from autonomous navigation to smart agriculture, and has the potential to make a substantial benefit across diverse fields.